This really looks very promising. Good find!

In the linked article they also clear the references to node (they call it elem):

# Also eliminate now-empty references from the root node to <Title>
while elem.getprevious() is not None:
    del elem.getparent()[0]

Would that make sense here too? I really don't know.

No, that makes not sense before, because we was use standard xml.etree.ElementTree instead lxml as suggested in the article.

As I understand, lxml is somehow extends ElementTree class from standard library. In particular, it adds links to parent nodes in childs. So we want to remove this links from cleared nodes during iteration, to tell gc free unused resources.

Now I added lxml module as suggested in the article because their iterparse method provides argument tag. It allows search nodes with given tag for iteration without traverse the whole tree. This sufficiently reduce execution time with the same dumpfile:

It is because we don't iterate over all unnecessary nodes as here. And it faster than current master implementation, when we iterate whole tree twice.

So, can we add lxml as dependency to cppcheck? It's is quite a popular module and I'm sure that most *nix users already have it pre-installed.

These changes reduce execution time and memory consumption in comparison with master. All tests will work if lxml would be added.

But in fact, this does not solve the problem with RAM completely. What will happen if we try to check a really large dump file? Consider 430+ Mb dump of jcdctmgr.c:

git clone https://github.com/mozilla/mozjpeg && cd mozjpeg
$INSTALL_PATH/bin/cppcheck -j4 --enable=all . --dump
du -h jcdctmgr.c.dump  # 431M

If we run misra.py with memory-profiler (mprof run misra.py jaricom.c.dump), we'll see something like this:

About 8 Gb RAM when loading variables and tokens during Configuration serialization.

What should we do about this problem? May be... check the size of XML dump before loading and, if it's requires more RAM than available, use iterparse in Configuration constructor? I suppose it will be very slow (need to check), but it doesn't kill the process at least.

Any suggestions?

I see. I did not have in mind that lxml is not used.
Maybe the memory consumption reduces to something acceptable when the constructor calls are delayed, like suggested in #2329 (Configuration for example).

IMHO it would be the best if users do not have to install extra python modules for using addons. That would make it more complicated to use the addons than it is now. Personally I would only use available modules for the addons. But I can not decide this.
What do you think about it @danmar

Maybe the memory consumption reduces to something acceptable when the constructor calls are delayed

Unfortunately, no. If we have no enough memory to load whole XML in memory, we'll die at this place. You can check it out on the large dump file from my post above.

IMHO it would be the best if users do not have to install extra python modules for using addons.

I agree, it can complicate installation process. Ok, I'll try to find other ways to do iterparse without lxml.

I agree, it can complicate installation process. Ok, I'll try to find other ways to do iterparse without lxml.

It is not a requirement to use xml.. feel free to suggest some alternative.

I think this sounds like a major refactoring that it's better to merge after the release.

Maybe the memory consumption reduces to something acceptable when the constructor calls are delayed

Unfortunately, no. If we have no enough memory to load whole XML in memory, we'll die at this place. You can check it out on the large dump file from my post above.

When using iterparse we do not need to load the whole XML into memory, so this additional optimization could help to further reduce the memory consumption I thought.
It seems like there are many ways to optimize the scripts and we need to implement several of them.
IMO using iterparse, like implemented in the first commit, was fine. That could be the first out of several steps optimizing the scripts.

IMO using iterparse, like implemented in the first commit, was fine. That could be the first out of several steps optimizing the scripts.

sounds ok to me.

@versat you was absolutely right to mention Configuration objects.

I rewrote iterative parser correctly (without third-party modules, correct resources cleanup) and, sigh, it wasn't solve this problem. With this parser we'll save only dump_file_size amount of RAM. This is quite insufficient in comparison of overall memory consumption. In particular, it saves 20 (of 400) Mb RAM with jaricom.c dump and ~400 (of ~10000) Mb with jcdctmgr.c dump.

So I was wondering what's really requires so much memory. Obliviously, the problem in serialized Python objects. So I use convenient pympler module to measure memory usage of particular objects. Here's some results with jaricom.c dump:

Append configuration 31954 Kb. New overall size: 0 Kb
Append configuration 31966 Kb. New overall size: 30024 Kb
Append configuration 31962 Kb. New overall size: 60048 Kb
Append configuration 31968 Kb. New overall size: 90046 Kb
Append configuration 31965 Kb. New overall size: 120072 Kb
Append configuration 32478 Kb. New overall size: 150065 Kb
Append configuration 32477 Kb. New overall size: 180610 Kb
Append configuration 31948 Kb. New overall size: 211140 Kb
Append configuration 31945 Kb. New overall size: 241142 Kb
Append configuration 31946 Kb. New overall size: 271144 Kb
Append configuration 31949 Kb. New overall size: 301145 Kb
Append configuration 31999 Kb. New overall size: 331142 Kb

Overall size: 361159 Kb
        All rawtokens: 1550 (5261 Kb)
        All tokens: 80192     (340218 Kb)
        All variables: 6230   (169743 Kb)
        All functions: 1156   (98174 Kb)
        All directives: 3972  (1978 Kb)

As we see, the most memory consumed by configurations list. So the solution seems pretty straightforward. If we see, that our dump files is pretty large, we should return iterator to configurations. The values for particular configuration would be parsed by demand.

The size of metadata from <dumps> nodes is insignificant. We could parse it in Configuration.__init__. And in constructor we should detect which way we will for parsing configurations: serialize all objects in memory (faster) or use iterative approach (slower, more resource-friendly). I suppose it could be done based on available amount of virtual memory.

Further experiments will take some time, I'll continue work on it.

P.S. Actually, I'm not interested at all in mozjpeg project. It was just random github repo, which crashed my small script which scraps github repos and try to use Cppcheck addons on them.

Very interesting, thanks for sharing these finds.
Did you test with Python 2 or Python 3, or even both? My experience is that Python 3 is faster and uses significantly less memory. So I am just curious about from which version these numbers come.
I do not expect that Python 3 solves any of the problems with the huge memory consumption, but it should at least use less memory.

I am not sure if it makes sense to implement different approaches for parsing the Configuration.
I fear that could have some drawbacks (like harder readability and maintainability, more error-prone), and I am also not sure if it is possible to tell which approach is the best by looking at the free virtual memory or so. But I could be wrong and this works great in the end :)

Did you test with Python 2 or Python 3, or even both?

I'm still using python 2.7.15 for all experiments to make sure that we have backward compatibility. Undoubtedly using latest python 3 versions will improve performance and optimize resources usage. Especially useful for us would be dataclasses (introduced in python 3.7). It would be interesting to explore other mechanisms like __slots__ and... may there is something else? This is a good place for further optimizations.

I'm not yet sure about configurations parsing, need to try that first and write tests later.

I finished iterative parser for Configuration objects. I got a significant reduction in memory consumption with same execution time in comparison with current master version.

Here are my results:

Environment: Debian 10.2, Python 3.7.3 and 2.7.16. Target machine: i5-3230M / 16 GiB DDR3-1600 RAM.

I use dump file jaricom.c described in previous posts. The similar results can be obtained with any dump file, e.g. addons test files or cppcheck sources.

@danmar @versat could you please review these changes? I also want to squash my commits to write a good commit message before merging if you accept it.

After these changes it also possible to process the huge dumps. Just look at these graphs given from handling jcdctmgr.c.dump:

I can't reproduce same with master, because I have no enough memory on my machine.

I have not looked too deeply into the code but from what I have seen, it looks fine.
Also, some tests I made revealed no issues. So I would give it a go as soon as 1.90 has been released.

So I would give it a go as soon as 1.90 has been released.

That is OK!

@danmar I suppose it can be merged now.

Let's try it!